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DETAILED ACTION 
Comments 

1 . The Amendments filed on February 20, 2007 have been entered and made of 
record. 

Claim Rejections - 35 USC §112 

2. The following is a quotation of the second paragraph of 35 U.S.C. 1 1 2: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

3. Claims 5, 6, 15 and 16 are rejected under 35 U.S.C. 112, second paragraph, as 
being indefinite for failing to particularly point out and distinctly claim the subject matter 
which applicant regards as the invention. 

Claim 5 recites the limitation "the first threshold value" in Iine2. There is 
insufficient antecedent basis for this limitation in the claim. Also, claim 5 is 
dependant upon cancelled claim 4. It has been treated as dependant upon claim 
1. 

Claim 6 is dependant upon claim 5. 

Claim 15 recites the limitation "the matrix" in line 3. There is insufficient 
antecedent basis for this limitation in the claim. However, it appears to be "the at 
least one segmentation matrix" and has been treated as such. 
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Claim 16 is dependant upon claim 15. 
Appropriate correction is required. 

Claim Objections 

4. Claim 1 5 is objected to because of the following informalities: "E(x,y);" in line 5 
should end with a period -- E(x,y). -. Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 1, 3, 9, 17 and 18 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Berube et al. (US 7,130,454) in view of de la Escalera et al. {"Road 
Traffic Sign Detection and Classification", IEEE Trans, on Industrial Electronics, Vol. 44, 
No. 6, Dec. 1997, pp 848-859). 

Re claim 1 , Berube et al. teaches a method for recognizing a road sign in a 
digital color image, where the road sign is associated with a shape template and at least 
one color criterion (Col. 7 lines 33-41 ), the method comprising the steps of: capturing a 
digital color image (FIGs. 1 and 2, "image acquisition stage 22 and video camera 40", 
Col. 7 lines 42-62); performing color segmentation on the digital color image to produce 
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at least one segmentation matrix ("blob", Col. 9 lines 24-57); correlating at least one 
region of interest within the segmentation matrix with a template matrix ("Reference 
Set", Col. 15 line 59 through Col. 16 line 27), where the template matrix is specific to a 
reference sign; and recognizing the image as containing the reference sign 
("Recognized", Col. 15 lines 21-38), responsive to the correlating step. 

As to claim 1, Berube et al. does not explicitly disclose wherein the performing 
color segmentation comprises comparing a plurality of color components at each of a 
plurality of pixel locations, within the digital color image to the color criterion, the color 
criterion comprising at least one relationship between two color components selected 
from the group of R, G, B. 

De la Escalera et al. teaches comparing a plurality of component at each of a 
plurality of pixel locations within the digital color image to the color criterion ("g(x, y)"), 
the color criterion comprising at least one relationship between two color components 
("color ratio") selected from the group of R, G, B (Section II 'Traffic Sign Detection', 
Subsection A 'Color Thresholding', page 849). 

Therefore, in view of de la Escalera et al., it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to modify Berube et al.'s 
method by incorporating the color thresholding, as taught by de la Escalera et al., which 
comprises a relationship between two color components selected from the group of R, 
G, B in order to achieve a system where special hardware is not used and the 
computational expense is minimum (Section II 'Traffic Sign Detection', Subsection A 
'Color Thresholding', page 849). 
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As to claim 3, Berube et al. does not explicitly disclose wherein the at least one 
color criterion comprises a plurality of color criteria and the performing step further 
comprises: setting each element of the at least one segmentation matrix to a first value 
if the corresponding pixel position in the digital color image matches any of the plurality 
of color criteria associated with the road sign. 

De la Escalera et al., further teaches the at least one color criterion {"g(x, y)") 
comprises a plurality of color criteria ("ki and k 2 ") and the performing step further 
comprises: setting each element of the at least one segmentation matrix (FIG. 1) to a 
first value {"ki) if the corresponding pixel position in the digital color image matches any 
of the plurality of color criteria associated with the road sign (Section II Traffic Sign 
Detection', Subsection A 'Color Thresholding', page 849). 

Therefore, in view of de la Escalera et al., it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to modify Berube et al.'s 
method by incorporating the method steps of setting each element of the at least one 
segmentation matrix to a first value k, if the corresponding pixel position in the digital 
color image matches any of the plurality of color criteria associated with the road sign in 
order to achieve the color thresholding using a 16-bit LUT and in real-time (Section II 
Traffic Sign Detection', Subsection A 'Color Thresholding', page 849). 

As to claim 9, Berube et al. does not explicitly disclose the segmentation matrix 
contains only binary values. 
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De la Escalera et al. further teaches the segmentation matrix contains only binary 
values (FIG. ^^ki and k 2 ", Section II Traffic Sign Detection', Subsection A 'Color 
Thresholding', page 849). 

Re claims 17 and 18, Berube et al. teaches a computer readable medium/system 
(FIG. 1, "system 20") having a computer program for recognizing a road sign in a digital 
color image, where the road sign is associated' with a shape template and at least one 
color criterion (Col. 7 lines 33-41), comprising: logic configured to capture/means for 
capturing a digital color image (FIGs. 1 and 2, "image acquisition stage 22 and video 
camera 40", Col. 7 lines 42-62); logic configured to perform/means for performing color 
segmentation on the digital color image to produce at least one segmentation matrix 
("blob", Col. 9 lines 24-57); logic configured to correlate/means for correlating at least 
one region of interest within the digital color image with a template matrix, where the 
template matrix is specific to a reference sign ("Reference Set", Col. 15 line 59 through 
Col. 16 line 27); and logic configured to recognize/means for recognizing the image as 
containing the reference sign, responsive to the correlation logic ("Recognized", Col. 15 
lines 21-38). 

As to claims 17 and 18, Berube et al. does not explicitly disclose wherein the 
logic configured to perform/means for performing color segmentation further comprises 
logic configured to compare a plurality of color components at each of a plurality of pixel 
locations within the digital color image to the color criterion, the color criterion 
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comprising at least one relationship between two color components selected from the 
group of R,G, B. 

De la Escalera et al. teaches the logic configured to perform color segmentation 
further comprises logic configured to compare a plurality of color components at each of 
a plurality of pixel locations within the digital color image to the color criterion {"g(x, y)"), 
the color criterion comprising at least one relationship between two color components 
("color ratio") selected from the group of R,G, B (Section II 'Traffic Sign Detection', 
Subsection A 'Color Thresholding', page 849). 

Therefore, in view of de la Escalera et al., it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to modify Berube et al.'s 
system by incorporating the color thresholding, as taught by de la Escalera et al., which 
comprises a relationship between two color components selected from the group of R, 
G, B in order to achieve a system where special hardware is not used and the 
computational expense is minimum (Section II 'Traffic Sign Detection', Subsection A 
'Color Thresholding', page 849). 

As to claims 19, 22 and 24, Berube et al. does not explicitly disclose the color 
criterion further comprising a first color component selected from R, G, B being greater 
than a first threshold value, the first threshold value expressed in terms of a second 
color component selected from R, G, B that is different than the first color component. 

De la Escalera et al. further teaches the color criterion further comprising a first 
color component selected from R, G, B being greater than a first threshold value, the 
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first threshold value expressed in terms of a second color component selected from R, 
G, B that is different than the first color component ("the pixels searched have a high 
red value with respect to their green and blue values." Section II Traffic Sign Detection', 
Subsection A 'Color Thresholding', page 849). 

7. Claims 2 and 13 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Berube et al. in view of de la Escalera et al. as applied to claim 1 above, and further in 
view of Laumeyer et al. (US 6,453,056): The teachings of Berube et al. modified by de 
la Escalera et al. have been discussed above. 

Berube et al. further teaches correlating the at least one submatrix ("ROI") with a 
template matrix ("reference set") to produce a correlation coefficient (Equation 7), where 
the template matrix is specific to a reference sign ("l R (Xj, y,)", Col. 15 line 59 through Col. 
16 line 27); and recognizing the image as containing the reference sign, based upon a 
comparison of the correlation coefficient and a correlation threshold value 
("predetermined thresholding value", Col. 16 lines 16-27). 

As to claim 2, Berube et al. modified by de la Escalera et al. fails to disclose 
extracting at least one region of interest from the at least one segmentation matrix to 
produce at least one submatrix containing at least one potential road sign. 

Laumeyer et al. teaches extracting at least one region of interest ("dynamically- 
sized region of interest (ROI)") from the at least one segmentation matrix to produce at 
least one submatrix containing at least one potential road sign (Col. 13 lines 12-45). 
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Therefore, in view of Laumeyer et al., it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to further modify Berube et al. 
and de la Escalera et al. by incorporating the method step of extracting at least one 
dynamically-sized region of interest from the at least one segmentation matrix to 
produce at least one submatrix containing at least one potential road sign in order to 
create a "sign list" which is used to correlate with a reference sign, wherein if it is 
confirmed that the portion of the image does not contain the sign, the ROI can be 
enlarged for further processing (Col. 13 lines 12-45). 

As to claim 13, Berube et al. further teaches calculating a two-dimensional 
correlation coefficient (Equation 7), which measures the correlation between the 
submatrix and a two-dimensional template matrix (Col. 15 line 59 through Col. 16 lines 
27). 

8. Claims 5 and 6 are rejected under 35 U.S.C. 1 03(a) as being unpatentable over 
Berube et al. in view of de la Escalera et al. as applied to claim 1 above, and further in 
view of Nakajima et al. (US 6,285,778). The teachings of Berube et al. modified by de 
la Escalera et al. have been discussed above. 

As to claim 5, Berube et al. modified by de la Escalera et al. fails to disclose the 
at least one color criterion is locally adaptive such that the first threshold value varies at 
each location within the template matrix. 
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Nakajima et al. teaches the at least one color criterion ("average threshold") is 
locally adaptive such that the first threshold value varies at each location within the 
template matrix ("window of mxn pixels", Col. 5 lines 10-20). 

Therefore, in view of Nakajima et al., it would have been obvious to one of 
ordinary skill in the art at the time the invention was made to further modify Berube et al. 
and de la Escalera et al. by incorporating the method step of thresholding the intensity 
of the pixels using the average threshold which is locally adaptive within the window of 
mxn pixels in order to achieve a better implementation when there exists a large 
brightness difference within the original image (Col. 5 lines 10-20). 

As to claim 6, Berube et al. modified by de la Escalera et al. fails to disclose the 
first threshold value at a location X(i,j) is the average of a square submatrix centered at 
X(ij). 

Nakajima et al. further teaches the first threshold value at a location X(i,j) is the 
average of a square submatrix ("window of mxn pixels") centered at X(i,j) ("when m = n", 
Col. 5 lines 10-20). 

i 

9. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Berube et 
al. in view of de la Escalera et al. as applied to claim 1 above, and further in view of 
Gilbert et al. (US 4,143,264). The teachings of Berube et al. modified by de la Escalera 
et al. have been discussed above. 
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As to claim 7, Berube et al. modified by de la Escalera et al. fails to disclose a 
second color criterion, where the first criterion is used in dim light conditions and the 
second criterion is used in bright light conditions. 

Gilbert et al. teaches a second color criterion ("higher threshold"), where the first 
criterion ("lower threshold") is used in dim light conditions ("low ambient light") and the 
second criterion is used in bright light conditions ("higher ambient light", Col. 2 lines 9- 
25). 

Therefore, in view of Gilbert et al., it would have been obvious to one of ordinary 
skill in the art at the time the invention was made to further modify Berube et al. and de 
la Escalera et al. by incorporating the higher and lower thresholds, as taught by Gilbert 
et al., used in low ambient light and higher ambient light to the color thresholding criteria 
of de la Escalera et al. in order to achieve an efficient sensitivity between natural 
sunlight and artificial light (Col. 1 lines 11-57). 

10. Claim 10 is rejected under 35 U.S.C. 103(a) as being unpatentable over Berube 
et al. in view of de la Escalera et al. as applied to claim 1 above, and further in view of 
Kim et al. (US 2001/0040982). The teachings of Berube et al. modified by de la 
Escalera et al. have been discussed above. 

As to claim 10, Berube et al. modified by de la Escalera et al. fails to disclose 
recursively removing any invalid row and any invalid column from the segmentation 
matrix, where an invalid row contains less than a first threshold value of a binary value 
and an invalid column contains less than a second threshold value of the binary value. 
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Kim et al. teaches recursively removing any invalid row and any invalid column 
from the segmentation matrix ("binary grid image"), where an invalid row contains less 
than a first threshold ("defined threshold") Value of a binary value and an invalid column 
contains less than a second threshold ("defined threshold") value of the binary value 
(Paragraphs [0028] and [0050]). 

Therefore, in view of Kim et al., it would have been obvious to one of ordinary 
skill in the art at the time the invention was made to further modify Berube et al. and de 
la Escalera et al. by incorporating the method step of recursively removing any invalid 
row and any invalid column from the binary grid image, where an invalid row/column 
contains less than a first/second threshold of the binary value in order to optimize the 
minimum area to become the shape of the object (Paragraph [0050]). 

1 1 . Claims 15 and 16, as understood, are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Berube et al. in view of de la Escalera et al. and Laumeyer as applied 
to claim 2 above, and further in view of Xia et al. (US 2001/0036300). The teachings of 
Berube et al. modified by de la Escalera et al. and Laumeyer have been discussed 
above. 

As to claim 15, Berube et al. modified by de la Escalera et al. and Laumeyer fails 
to disclose scanning the matrix for an element with value 1 at position E(x,y); and 
performing a depth-first-search to find all elements with value 1 that are connected to 

E(x,y). 
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Xia et al. teaches scanning the matrix for an element with value 1 ("binary 
transitions") at position E(x,y) ("P"); and performing a depth-first-search to find all 
elements with value 1 that are connected to E(x,y) ("Rules", Paragraphs [0055]-[0069]). 

Therefore, in view of Xia et al., it would have been obvious to one of ordinary skill 
in the art at the time the invention was made to further modify Berube et al., de la 
Escalera et al. and Laumeyer by incorporating the method step of scanning the matrix 
for an element with value 1 at position E(x,y); and performing a depth-first-search to find 
all elements with value 1 that are connected to E(x,y), using the algorithm an rules as 
taught by Xia et al. in order to remove foreground pixels from background pixels to 
efficiently reduce the number of false connections (Paragraphs [0070] and [0071]). 

As to claim 16, Berube et al. modified by de la Escalera et al. and Laumeyer fails 
to disclose setting the element at position E(x,y) to a unique identifier; and setting the 
value of the connected elements to the unique identifier. 

Xia et al. teaches setting the element at position E(x,y) to a unique identifier ("1"), 
and setting the value of the connected elements to the unique identifier (FIGs. 4 and 5, 
Paragraph [0071]). 

Allowable Subject Matter 

12. Claims 11, 12, 20, 21, 23 and 25 are objected to as being dependent upon a 
rejected base claim, but would be allowable if rewritten in independent form including all 
of the limitations of the base claim and any intervening claims. 
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1 3. The following is a statement of reasons for the indication of allowable subject 
matter: The closest prior art made of record fails to disclose or suggests the creation of 
the validating row and column vectors and the correlation between template vectors, the 
normalization of the submatrix and color criterion being expressed as the difference 
between two color components compared with the difference of one color component 
with the remaining color component, when a R, G, B color space is used. 

Response to Arguments 

Objections to the Drawings 

14. Figure 3 has been amended to replace reference character " 1 05" by reference 
character "310" and reference character "210" by reference character "31 1". Therefore, 
the objections have been removed. 

Figure 5 has been amended to replace reference character "516" by reference 
character "513", and to correct the subscripts of reference characters "515" and "516" to 
" X\", and "A', 2 ", respectively. Therefore, the objections have been removed. 

Figure 9 has been amended to replace reference character "204" by reference 
character "203". Therefore, the objection has been removed. 

Figure 10B has been amended to include reference character "1005". Therefore, 
the objection has been removed. 

Figures 10D and 10E have been amended to include reference character "1007". 
Therefore, the objections have been removed. 
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Figure 10F have been amended to include reference character "1007" 
appropriately and replace reference character "1007" by reference character "1008". 
Therefore, the objections have been removed. 

Objections to the Specification 
a. Informalities 

1 5. Paragraph [01 5] has been amended to appropriately recite "FIGs. 1 0A-G" in line 
1 . Therefore, the rejection has been removed. 

Paragraph [021] has been amended to replace reference character "108" in line 6 
with reference character "106". Therefore, the rejection has been removed. 

Paragraph [022] has been amended to replace reference character "109" in line 1 
with reference character "106". Therefore, the objection has been removed. 

Paragraph [025] has been amended to replace reference character "110" in line 4 
with reference character "106". Therefore, the objection has been removed. 

Paragraph [026] has been amended to replace reference character "1 1 1" in line 1 
and reference character "112" in line 3, with reference character "106" on both 
instances. Therefore, the objections have been removed. 

Paragraph [027] has been amended to replace reference character "113" in line 1 
and reference character "114" in line 2, with reference character "106" on both 
instances. Therefore, the objections have been removed. 

Paragraph [042] has been amended to replace reference character "309" in line 4 
with reference character "31 1". Therefore, the objection has been removed. 
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Paragraph [050] has been amended to replace, "S ] =E(x ] 0 ,x ] 0 ;\,n)" in lines 3-4 
with u S l = E{x\,xlXn)" ■ Therefore, the objection has been removed. 

Paragraph [056] has been amended to replace reference character "604" in line 1 
with reference character "605". Therefore, the objection has been removed. 

Paragraph [058] has been amended to replace reference character "604" in line 1 
with reference character "608" and to replace reference character "605" in line 2 with 
reference character "608". However, from the Drawings it appears to be referring to 
"step 606" in line 1 and "step 608" in line 2. Therefore, the objections are maintained. 

Paragraph [058] has been amended to replace reference character "607" in line 3 
with reference character "608". Therefore, the objection has been removed. 

Paragraph [059] has been amended to replace reference character "608" in line 
1 , and reference character "609" in line 3, with reference characters "609" and "610", 
respectively. Therefore, the objections have been removed. 

Paragraph [060] has been amended to replace reference characters "604" with 
reference character "607" in lines 1 (both instances) and 3, and replace reference 
character "610" with reference character "612" in line 7. Therefore, the objections have 
been removed. 

Paragraph [068] has been amended to replace reference character "708" in line 
4, and reference character "709" in line 5, with reference characters "709" and "708", 
respectively. Therefore, the objections have been removed: 
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Paragraph [095] has been amended to clearly define the term OCR by adding 
"(Optical Character Recognition" in lines 1-2. Therefore, the objection has been 
removed. 

b. Antecedent Basis 

16. Proper antecedent basis for claim 6 has been found on Paragraph [073] as 
asserted by the Applicant on Page 13 of the First Response and Amendment. 
Therefore, the objection has been removed. 

Claim 1 1 has been amended to properly define "a validating column/row vector", 
which finds support on Paragraph [055]. Therefore, the objection has been removed. 

Claim Objections 

17. Claim 8 has been cancelled. Therefore, the objection has been removed. 
Claim 15 has been amended to correct sentence grammar in line 3 by replacing 

"the matrix for a element" with "the matrix for an element". Therefore, the objection has 
been removed. 

Claim 17 has been amended to correct sentence grammar in line 7 by replacing 
"logic correlate at least one region" with "logic configured to correlate at least one 
region". Therefore, the objection has been removed. 

Claim 18 has been amended to correct sentence grammar in line 3 by replacing 
"the method" with "the system". Therefore, the objection has been removed. 
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Claim Rejections under 35 U.S.C. $ 101 

1 8. Applicant has amended the specification to delete the words "electromagnetic 1 
and "infrared" from Paragraph [026]. Therefore, the rejection has been removed. 

Claim Rejections under 35 U.S.C. §112 

19. Claim 2 has been amended to recite "the at leas one segmentation matrix" in line 
3, and "the reference matrix" in line 6. Therefore, the rejection has been removed. 

Claim 3 has been amended to recite "the at least one segmentation matrix" in 
line 3", and proper antecedent basis has been found in claim 1 for "the performing step". 
Therefore, the rejection has been removed. 

Claim 5 has been amended to recite "the template matrix" in lines 2-3. 
Therefore, the rejection has been removed. 

Claim 6 is dependant upon claim 5. Therefore, the rejection has been removed. 

Claim 9 has been amended to recite "the segmentation matrix" in line 1 . 
Therefore, the rejection has been removed. 

Claim 10 has been amended to recite "the segmentation matrix" in lines 2-3. 
Therefore, the rejection has been removed. 

Claim 12 has been amended to depend upon claim 2. Therefore, the rejection 
has been removed. 

Claim 13 has been amended to depend upon claim 2. Therefore, the rejection 
has been removed. 

Claim 14 has been cancelled. Therefore, the rejection has been removed. 
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Claim 1 5 has been amended to depend upon claim 2. However, this correction 
does not clarify the antecedent basis for the claim limitation "the matrix" in line 3. 
Therefore, the rejection is maintained. 

Claim 16 is dependent upon claim 15. Therefore, the rejection is maintained. 

Claim Rejections under 35 U.S.C. $ 102 

20. Applicant's arguments with respect to claims 1, 2, 7, 13, 14, 17 and 18 have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Rejections under 35 U.S.C. $ 102 

21 . Applicant's arguments with respect to claims 3-6, 8-12, 15 and 16 have been 
considered but are moot in view of the new ground(s) of rejection. 

Conclusion 

22. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. Imao et al. disclose an Image Separator for Color Image 
Processing, Rosser et al. disclose a Pattern Recognitions System Employing Unlike 
Templates to Detect Objects Having Distinctive Features in a Video Field, Pdchmuller 
disclose a Method for the Displaying Information in a Motor Vehicle and Piccioli et al. 
disclose a Robust Road Sign Detection and Recognition from Image Sequences. 

Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
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§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jose M. Torres whose telephone number is 571-270- 
1356. The examiner can normally be reached on Monday thru Friday: 8:00am - 
4:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jingge Wu can be reached on 571-272-7429. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

JMT 

04/12/2007 




